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Introduction 



The process of problem-based learning starts with a problem, consisting of a 
description of a set of phenomena in need of explanation. A groupof eight to ten 
students, discusses the problem and tries to explain the phenomena in terms of 
underlying processes, principles or mechanisms (Schmidt, 1983). Duri g this 
discussion issues emerge needing further exploration. These learning issues are the 
starting point for students' learning activities and serve as a guide -for studying ' 
literature or other educational sources. Thus, student-generated learning issues are 
topics that each tutorial group decides are prerequisites to a better understanding of 
the problem under discussion. Each student individually seeks information jegarding 
the learning issues identified. Two days later the group meets again and each 
participant reports what he or she has found. In this second meeting an attempt is 
made to integrate information collected and to draw conclusions. Each tutorial group 
is guided by a tutor, whose main task is facilitating group interaction. 

The emphasis in problem-based learning is on the active acquisition of 
knowledge. Students themselves more or less define the content to be mastered. An 
often mentioned advantage of this student-centered learning approach is that students 
develop self-directed learning skills (Barrows, 1985; Blumberg & Michael, 1991). 
However, this approach to learning and instruction also is a source of concern to both 
teachers and students. Teachers worry that 'important' information will not be 
studied by students in a problem-based curriculum (Coulson & Osborne, 1984). 
Students, at least initially are thought to be unable to generate all the appropriate 
learning issues and would not identify all the 'required' knowledge (Blumberg, 
Michael & Zeitz, 1990). 

Students are responsible for their own learning, but teachers also have their 
responsibility for developing the problems that serve as stimuli for self-directed 
learning. Problem design is a painstaking activity since problems should direct 
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students into specific content domains. Unlike in traditional educational systems, 
teachers have no direct influence on students' learning because students generate 
their own learning issues. In cases where students do not generate those learning 
issues as intended by teachers, students will not reach the intended faculty objectives. 
Faculty objectives define the subject-matter that students need to master. If students 
do not identify the intended faculty objectives, students' learning activities do not 
cover the intended content-coverage. Therefore, information is needed about the 
correspondence between faculty objectives and students' learning issues. The main 
focus of this paper is to present a method for collecting information about the match 
between students' learning issues and teachers' objectives. These data can provide 
insight into the learning issues which will arise from problems and the degree to 
which the intended objectives are reached by students. 



Method 



Subjects. This study was conducted at the Medical School of the University of 
Limburg, The Netherlands. The first four >2 ars of the problem-based curriculum are 
structured in units, a series of six-week courses. In this study 82 second year students 
participated of the 1989-90 academic year divided in 12 tutorial groups. These 
students attended a six-week course on normal pregnancy, delivery and normal 
development of children and adolescents. The unit consisted of problems concerning 
topics such as childbirth, childbed, psycho-motor development and psychosexual 
development Students were randomly assigned to tutorial groups. 

Procedure. During this unit learning issues generated by the students were 
collected. Each tutor was asked to record the learning issues generated by his or her 
tutorial group for each problem. A few weeks after the end of the unit, raters were 
asked to judge the correspondence between faculty objectives and students' learning 
issues. The raters were students who participated in the tutorial groups of this unit. 
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Student-ratets seem to be the closest source because they are familiar with the unit 
content. Moreover, pilot-studies revealed that results from teacher-raters and student- 
raters were quite similar. The unit consisted of 12 problems and 61 faculty 
objectives, three to nine objectives for each problem, with an average of 5.1. These 
faculty objectives were in teachers' mind while constructing the 12 problems. 

Each tutorial group generated 1 to 13 learning issues for each problem, with an 
average of 3.6. In order to match teachers' objectives with students' learning issues 
for one problem within one tutorial group, raters had to make about 18 (3.6 * 5.1) 
comparisons. Consequently, if each rater would have to judge 12 problems and 12 
tutorial groups, about 2500 comparisons had to be made. In order to reducethe 
number of comparisons to be made, a design was used in which raters were nested 
within problems and within tutorial groups. This design is shown in Table 1. 24 
raters, divided into 12 pairs, were asked to judge to what extent the learning issues of 
a specific tutorial group had matched the objectives intended by the teachers for a 
particular problem. 



Table 1 

Distribution of raters among problems and tutorial groups 



Tutorial groups 



Problems 


1 


2 


3 


12 


1 








^23^4 


2 


^23»24 






^21,22 


3 


^2 1,22 


^2*24 


• • 




• • 

12 
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Analysis. The task for the raters consisted in comparing each learning issue 
with each faculty objective within one problem and one tutorial group. In order to 
illustrate this, the intended faculty objectives and the learning issues generated by 
one tutorial group for Problem 10 will be shown. 

Problem 10: Ellen 

Last few years Ellen has grown tall very fast. She has always been a tall girl, but 
at an age of 11 years arid 5,4 feet in height she rises head and shoulders above 
her age group. People do always take her to be older, which sometimes becomes 
wearisome. What will become of her? She still has not reached her age of 
puberty 



Teachers had five faculty objectives 
in mind while developing this 
problem: 



One tutorial group had generated 
five learning issues: 



1 Normal rates of child growth 

2 Normal stages in secondary 
sexual characteristics 

3 Endocrine control processes of 
growth 

4 Psychological effects of being 
extremely taller compared to the 
age group 

5 Diagnostic procedures to predict 
ultimate height 



1 Normal pattern of growth rate 

2 Which factors do influence 
growth, which hormones are 
concerned with growth and what 
are their effects? 

3 Physical changes during puberty 
and possible explanations 

4 Diagnostic possibilities with 
regard to growth 

5 Medical intervention of abnormal 
growth 
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Since Problem 10 in this unit contained five faculty objectives and one tutorial 
group had generated five learning issues, raters had to make 25 comparisons for this 
problem. For each comparison raters had to judge whether a particular learning issue: 
(1) definitely corresponds with a particular faculty objective or (2) that learning issue 
definitely does not correspond with that faculty objective. In Table 2 an example is 
shown of two raters judging Problem 10, containing five faculty objectives stated on 
the horizontal axis, with one tutorial group who had generated five learning issues, 
stated on the vertical axis. If one rater judged one or more learning issues 
corresponding to one faculty objective, then this particular objective was judged as 
'identified' by the tutorial group. As shown in the lowermost row in Table 2, rater 1 
judged four out of five objectives a$ identified and rater 2 judged three out of five 
objectives as identified by student learning issues. Both raters agreed that objective 
1, 3 and 5 were definitely identified by the students and that objective 4 was 
definitely not identified by the students. Both raters disagreed whether objective 2 
was identified by this tutorial group. A faculty objective was scored as identified if 
both raters agreed. If both raters agreed that the faculty objective is definitely not 
identified, then it is assumed by the researchers that the students failed to study the 
content domain specified for this objective. 
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Table 2 



An Example of the Ratings of Two Raters for One Problem and One Tutorial Group 



Raters 








1 




2 






r acuity objectives 




Faculty objectives 


Issues 


1 2 3 4 5 


Issues 


1 2 J 4 5 


i 


+ 


1 


+ 


2 


+ . 


2 




3 




3 




4 




. 4 


. - + 


5 




5 






+ + + . + 







Results 



The interrater-agreement for each pair, varied between 64.2 percent and 85.0 
percent, with an average of 76.9 percent. This implies that the raters disagreed about 
23.1 percent of the faculty objective whether they were reflected by students' 
learning issues. Since these percentages are overestimated, because of agreement 
based on chance, kappa-coefficients were computed for each pair of raters. The 
kappa-coefficient takes into account the expected proportion of agreement that would 
occur if assignments were made by chance alone and varies between values lower 
than zero and one. Hie kappa-coefficient for the paira of observers, varied between 
.22 and .57, with an average of .35. The average percentage of .35 demonstrates a 
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fair agreement between the raters (Landis & Koch, 1977). These results seem to 
support the reliability of the match procedure. Other studies (Dolmans, Gijselaers, 
Schmidt & Van der Meer, 1992) revealed similar findings, whereas the average 
kappa-coefficient was .45, demonstrating a moderate agreement. 

In order to assess whether students identify the intended faculty objecaves, 
information will be presented about the degree to which faculty objectives are 
explored by student-generated learning issues. The average percentage of overlap for 
each problem and each tutorial group is shown in Table 3. The cells in this table 
show the percentage of faculty objectives identified by students* learning issues. Hie 
mean percentage of overlap for each tutorial group varied between 51.9 and 73.9. 
This percentage does not differ across tutorial groups (F(ll,141)=.87, p=.569). The 
correspondence between faculty objectives and student-generated learning issues for 
each problem varied between 33.3 and 77.0 percent and differed across problems 
(F(ll,141)=2.99 /x.001). Hie average percentage of overlap for the 12 problems 
together was 62.4. In other words, an average of 62.4 percent of the faculty 
objectives were definitely identified by the 12 tutorial groups. This implies that an 
average of 37.6 percent of the faculty objectives were either definitely not identified 
by the students or the raters did not agree whether these faculty objectives were 
identified. Other studies (Dolmans, Gijselaers, Schmidt & Van der Meer, 1992) 
revealed a quite similiar average percentage of overlap of 64.2. 
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Table 3 

Percentage of teachers' objectives matched by students' learning issues for each problem and each 
tutorial group 



Tutorial groups 



r ruuicms 


* 

I 


2 


3 


4 


5 


6 


7 


8 


9 


10 


11 


12 


Mcai 


i 


89 


67 


67 


100 


67 


100 


100 


67 


33 


78 


89 


67 


77 n 


2 


100 


100 


29 


29 


100 


.71 


71 


43 


86 


86 


86 


43 


70.3 


,3 


25 


25 


75 


25 


75 


75 


50 


100 


100 


100 


100 


100 


70.8 


4 


67 


33 


44 


56 


11 


78 


67 


56 


67 


22 


78 


22 


50.1 


5 


67 


100 


0 


100 


83 


67 


50 


• * 


50 


67 


50 


83 


652 


6 


67 


67 


67 


67 


33 


100 


33 




100 


33 


67 


67 


63.7 


7 


75 


75 


75 


75 


100 


50 


50 


25 


75 


75 


100 


75 


70.8 


8 


100 


100 


67 


67 


100 


67 


67 


100 


33 


67 


100 


0 


72.3 


9 


40 


60 


80 


40 


40 


60 


20 


60 


60 


20 


80 


40 


50.0 


10 


75 


50 


50 


100 


100 


75 


75 


50 


50 


25 


25 


75 


62.5 


11 


75 


50 


50 


75 


75 


50 


75 


75 


75 


50 


75 


25 


62.5 


12 


0 


67 


67 


33 


67 


33 


0 


33 


33 


0 


33 


33 


33.3 


Mean 


65.0 


66.2 


55.9 


63.9 


70.9 


68.8 


54.8 


60.9 


63.5 


51.9 


73.6 


52.5 


62.4 



" Two «®* «* because two tutorial groups omitted one problem 
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Conclusion and discussion 

The results demonstrate that the method is useful in identifying ineffeotive 
problems, that is problems which do not lead students to generate the appropriate 
learning issues. In this study, for instance, Problem 12 seems to be an ineffective 
problem taken into account that only 33.3 percent of the intended objectives is 
identified by students' learning issues. Hiis bw percentage of overlap, however, may 
be due to end-of-unit-effects, such as decreasing motivation and lack of time because 
students are preparing their end-of-unit examination. Problems 4 and 9 also seem to 
be ineffective since the average percentage of overlap between the faculty objectives 
and students' learning issues is about 50 percent. The average percentage of content 
coverage for the unit as a whole is 62.4 (SD-26.6). Consequently/approximately 60 
percent of the faculty objectives are identified by students' learning issues. Similar 
findings are reported in other studies. For example, Tans, Schmidt, Schade-Hogeveen 
and Gijselaers (1986) who conducted a study concerning the correspondence between 
teachers' objectives and students' learning issues reported a mean percentage of 68.3 
(SD'24,6). Coulson and Osborne (1984) concluded that at the course level, tutorial 
groups succeeded in identifying all of the faculty-generated faculty objectives. 
However, each faculty-identified learning issue was generated on the average by five 
out of twelve tutorial groups (42 percent). 

The assumption underlying the relationship between a probfe - uid certain 
learning issues or course objectives is that each problem leads to formulate unique 
learning issues or objectives. However, it is questionable whether this assumption is 
valid. Analyzing the data concerning this issue revealed that students only in minor 
cases generated learning issues wliich were intended to be generated for other 
problems. Moreover, tutors remarked that most of these learning issues were not 
initially derived from the problem at hand, but were repeated because students did 
not master that issue and needed to spend some more study time. 



9 

ERIC 



II 



Dolmans, et al. j j 



Although problems are the starting point for students' learning activities and 
determine the generation of learning issues, it should be noted that it is not sure that 
students actually pursue these learning issues in all cases. Students will not employ 
certain learning activities related to a particular problem when thjy have already 
studied this material before. Students can also decide it to be more relevant to spend 
time on other issues. Moreover, it is possible that students pay a lot of attention on 
an issue during the discussion in the tutorial group and decide not to define this 
learning issue as a guide during independent study. This issue, although not 
formulated as a learning issue, might be dealt with adequately according to students' 
perceptions. On the other hand, it seems clear that during independent self-study and 
during, discussion in the tutorial group, students frequently learn about other related 
topics that go beyond student-generated learning issues. In summary, student- • 
generated learning issues define what students are going to study, but it is not sure 
that they will really lead student learning. Learning issues are only possible 
reflections of student learning. Although this question may be raised, preliminary 
results from a study conducted by Tans et al. (1986) revealed that learning issues are 
valid indicators for students' learning activities, since the correlation coefficient 
between the frequency of generating students' learning issues and students' 
achievement on test items corresponding with those issues was .42. 

Furthermore, it should be noted that the correspondence between faculty 
objectives and student-generated learning issues is quite difficult to judge since a 
learning issue is arisen from the discussion in the tutorial group that is not available 
to the raters. This lack of background information makes it quite difficult in some 
cases to judge to what extent a learning issue is relevant to a particular objective, 
although the students involved probably exactly know what is intended. The 
agreement between the raters, however, was fairly high indicating that in most cases 
the raters were able to make these judgments, without additional context information. 
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Another deficiency of the procedure presented to compare students* faculty 
objectives with student-generated learning issues is that, although it provides a means 
of detecting ineffective problems, it does not contain information about the nature of 
the shortcomings of these problems. In order to provide information about 
improvements it is necessary to identify features determining problems' quality. 
However, until now little is known about the criteria along which problems should be 
constructed. The effectiveness of a problem is supposed to be related to students' 
prior knowledge about the subject-matter, linkage to students' future profession, the 
length of the text and the presence of structuring remarks (Bouhuijs & Gijseiaers, 
1987; Majoor, Schmidt, Snellen-Baiendong, Moust, & Stalenhoef-Halling, 1990). 

Furthermore, research is needed about the nature of learning issues generated by 
students that do not correspond with teachers' objectives. Additional information 
should be collected in order to assess the appropriateness of these learning issues. 
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